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FOREWORD 



The Leadership and Management Technical Area of the Army Research Insti- 
tute for the Behavioral and Social Sciences (ARI) conducts research address- 
ing the issues of Army leader assessment, training, and performance. Since 
nearly 75% of new officers commissioned each year are trained through the 
Army Reserve Officers' Training Corps (ROTC), research on the ROTC training 
program has great potential for enhancing officer performance. 

In 1983, the Deputy Chief of Staff for ROTC initiated the Achievement 
Testing Program to assess the levels of basic skills of ROTC cadets with the 
objective of developing standardized diagnostic and selection procedures, 
ARI was asked to perform the data analysis. 

This report describes the results of the analyses in terms of the distri- 
bution of achievement test scores for Military Science I (MS I) and MS IV 
classes as a whole and for subgroups of cadets. The report also indicates 
additional types of information needed for ROTC to formulate diagnostic and 
selection standards as well as the organizational, social, and political 
issues that must be addressed in this process. 




EDGAR M. JOHNSON 
Technical Director 
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ROTC ACHIEVEMENT TESTING PROGRAM: SCHOOL YEARS 1983-1985 



EXECUTIVE SUMMARY 



Requirement : 

The 1978 report of the Review of Education and Training of Officers (RETO) 
recommended a standardized screening procedure to be established for precommis- 
sioning programs. The Deputy Chief of Staff for ROTC (DCSROTC), Training and 
Doctrine Command (TRADOC), initiated a project to assess basic skills of ROTC 
cadets • The ODCSROTC requested the Army Research Institute (ARI) to analyze 
achievement test data for School Years 1983-85 and to report findings that 
might be used by ROTC to develop diagnostic and screening standards. 



Procedure: 

ROTC administered the Missouri College English Test, the Nelson-Denny 
Reading Test, and the Stanford Achievement Test: High School Mathematics 
Test to Military Science I (MS I) and MS IV cadets* Nearly the entire popu- 
lation of MS IV cadets participated, while thft participation rate for MS I 
was a little better than 50%, However, for /xll classes tested, the propor- 
tional representation by region and gender closely approximated the percent- 
age breakdowns for these subgroups of cadets reported by the ROTC Headquarters 
for the respective school years and MS classes • 

This report is based on data from School Years (SYs) 1983-84 and 1984-85. 
The distributions of test scores were examined within each MS class by gender, 
racial/ethnic group, and academic major. 



Findings : 

Females scored higher than males in English; males scored higher than 
females in mathematics; and there were no substantial gender differences in 
reading scores. However, on all tests, the ethnic minority groups scored 
lower than the white group; the differences were substantial for the black 
and Hispanic groups. The average scores of MS I cadets were consistently 
lower than MS IV scores possibly due to attrition of lower achieving students, 
selection applied at contracting between MS II and MS III, and/or greater 
experience and education of MS IV cadets. Based on scores that might be used 
as possible cut scores for MS IV, the analyses identified about 2C% of MS I 
cadets who might profit from remedial training in English and reading. 
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ROTC Achlevemeat Testing Program: 
School Years 1983 - 1985 



Backgrouad 



An important aspect of precommlssloalng officer traiaing is to ensure that 
all aewly-commissioaed Army officers have attained aa adequate level of basic 
skills. The Army Reserve Officers • Training Corps (ROTC) produces nearly 75% 
of new officers commissioned each year, and this production mission is 
expected to increase in the coming years. The basic skills assessment and 
development effort in the ROTC program could make an important contribution to 
ROTC training effectiveness as well as officer utilization. 

The 1978 report of the Review of Education and Training of Officers (RETO) 
recommended that a standardized 'Uethod be established to screen and evaluate 
officer potential of candidates for precommissioning programs. The 
recommendation was followed by the Precommissioning Assessment System (PAS) 
(U.S. Department of the Army, 1978) which identified five components 
considered to be important to military leadership: academic, physical, 
medical, psychological, and leadership potentials PAS also established 
initial guidelines for assessing each of these components for the purpose of 
admitting, or contracting, cadets into ROTC Advanced Course. 

The only standard currently used to assess cadets' academic competence 
for commissioning is a college degree and grade point average of 2.0 or above. 
However, grading standards vary greatly among universities and colleges. 
Thus, a need for further efforts in refining evaluation standards was 
reiterated in the 1984 report of Army Science Board Summer Study (U.S. 
Department of the Army, 1984). 

Responding to this need, the Deputy Chief of Staff for ROTC (DCSROTC), 
the Training and Doctrine Command (TRADOC), initiated a project in 1983 to 
establish diagnostic and screening standards for standardized achievement 
tests. The plan was to administer achievement tests during threr^ consecutive 
school years to ROTC cadets in Military Science I (MS I) and Military Science 
IV (MS IV) classes. Based on the configuration of the scores obuiined from 
this initial phase of the project, ROTC was to formulate a standardized method 
of assessing and developing basic skills. ODCSROTC requested the krrjxy 
Research Institute (ARI) to analyze the achievement test data for School Years 
(SY) 1983-86 to provide a description of cadet performance which would be used 
by ROTC in setting diagnostic and screening standards. 
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Objectives of the Achlevemeat Testlag Program 



The Achievemeat Testing Program was designed to serve tuo purposes: 

1. To establish guidelines for diagnosing basic skills deficiencies 
among cadets in the ROTC Basic Course (MS I and II) for the 
purpose of directing those with marginal or poor performance to 
receive remedial training, 

2* To set criteria for proficiency in reading, writing, and 

mathematics for admissions to the Advanced Course of ROIC (MS 
III and IV) and for commissioning. 

The immediate objective for the achievement testing in school years 1983 
through 1986 was to establish an empirical basis for formulating preliminary 
standards for diagnosis, screening, and commissioning which might be adjusted 
when test results from subsequent years were examined. 

Purpose of Report ' 

The purpose of this report is to present results from the achievement test 
data analysis. It will describe test performance of ROTC cadets as a whole, 
comparisons among gender and ethnic subgroups, and how many cadets passed 
various score points. 

Setting selection standards is a highly complex process. Besides the test 
scores, many other issues must be taken into consideration such as the 
relationships between basic skills and officer job requirements, the number of 
new officers needed, resources needed to continue testing, and the social/ 
political implications of using cut scores for selection. Much of the 
information needed to address these issues is currently lacking, requiring 
further empirical research. 

The results presented in this report will provide one of numerous pieces 
of information required, namely the test performance of ROTC cadets. However, 
they alone will not be sufficient to specify sound guidelines for selection or 
diagnostic procedures. 



Procedure 

Achievement Tests Used in the Project 

ROTC selected three commercially available achievement tests to be used 
for this project. Each test provides national "norms," i,e., performance of a 
large sample of students across geographical areas, as a basis for comparison. 

Missouri College English Test (Callis & Johnsoc, 1965). This test was 
designed to assess knowledge of "the mechanics and effectiveness of written 
expression" (Callis & Johnson, 1965, p. 3). It is a 90-item multiple-choice 
test, requiring no actual writing. The test was standardized based on data 
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from college freshmea atteadlag uaiversities aad two- and four-year colleges, 
both private aad public, from all regions of the Uaited States, during the 
early 1960s. 

T%o issues need to be addressed ia usiag this test: (1) The natioaal 
Qorm was established for college freshmen only since the test was intended as 
a diagnostic tool to identify freshman students in need of remedial training 
in writing. When the test is administered to more advanced college students 
in higher college grades, as in the ROTC Achievement Testing Program, the 
interpretation of raw scores according to the national norms may be less 
accurate. Older students may be likely to score higher, due to greater 
experiences and learning opportunities and/or student attrition, part of 
which may be based on lack of basic skills. 

(2) The original norming sample was adequate in number and regional and 
type-o£-school representation. However, this was the only standardization 
performed for this test on a national sample. Since th^n, educational 
achievement levels have shifted (McGeever, 1983). It may, or may not, be 
accurate to judge writing abilities of the 80s students (even college 
freshmen) against the performance of freshmen in the early 60s. As described 
iu the subsequent section, these problems were partially dealt with by 
computing ROTC norms based on the data collected for this project. 

The Nelson-Denny Reading Test (Brown, Bennett, & Hanna, 1981). This 
test provides assessment of vocabulary knowledge, reading comprehension, 
and reading rate. The two subtests (Vocabulary, Comprehension) can be used 
selectively to diagnose students* weak areas. Scores from both subtests 
combined (Total) indicate their overall reading achievement. 

The national norms for this test were developed in the late 70s, based on 
scores from students in 9th through 16th grades (high school and college). 
Separate norms are published for each of the 8 grades included, so that, for 
example, a college senior's score can be compared to a national sample of 
college seniors. Interpreting the ROTC students* performance against the 
national norm is more accurate with this test than with the Missouri English 
test, because the standardization was done fairly recently and separate norms 
for each college grade are available. 

Stanford Achievement Test; High School Mathematics Test (Gardner, Merwin, 
Callis, & Madden, 1965). This test consists of three parts: a Numeric 
Competence subtest, which assesses students* general numeric and arithmetic 
capabilities; a Mathematics Part A subtest, which contains problems covered in 
high school elementary algebra and geometry textbooks, and a Mathematics Part 
B subtest, which tests knowledge of advanced algebra, trigonometry, and some 
new mathematics concepts from high school textbooks. For the purpose of 
diagnosis for remediation, the test writers suggest using the Numeric 
Competence subtest score. 

The two problems mentioned with reference to the Missouri English test 
also apply to this test. The test was standardized based on students in high 
school grades (9th through 12th); the reference group which is closest to ROTC 
cadets is the college preparatory, high school seniors. In addition, the 
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norming data were collected in the early 60s. Thus, iaterpretiag the 
performance of ROTC cadets, especially college juniors and seniors, according 
to these norms may not be very accurate. 

Reliability and validity of the achievement tests . The reliability of 
these tests reported in the manuals are all in the .908 range and indicate 
adequate stability of scores on these tests. Table 1 presents the reliability 
coefficients and methods of deriving them for each test. The content 
validity, i.e., the degree to which test items cover the subject matter, was 
investigated by test developers for each test and reported to be satisfactory. 
However, the question of how accurately these tests may predict college 
graduates* job performance, specifically in the military setting, has not been 
addressed empirically. 



Table 1 

Reliabilities of Achievement Tests^ 



Test 



Reliability 
Coefficient 



Sample 
Grade Number 



Method 
Used 



Missouri College 
English Test 



Nelson-Denny 
Reading Test 
(Total scores) 



.94 



.93 



13th 12,580 



13th & 
14 th 



239 



Split-half 
corrected by 
Spearman-Brown 

Alternate forms 
correlations 



S tanf ord-Ma thema tics 

Numeric Competence .92 
Mathematics Total .94 



12th 



4,897 



Split-half 
corrected by 
Spea rman-Br o wn 



* Extracted from Callis and Johnson (1965) for English, Brown et al. (1981) 
for reading, and Gardner et al. (1965) for mathematics tests. 



Test Administration Procedure 



The ROTC-wide administration of the three achievement tests for the 
purpose of developing diagnostic and screening standards was carried out 
over three consecutive school years beginning with SY 83-84. During SY 83-84, 
the English and the reading tests were administered in tha spring semester to 
MS IV classes of mostly juniors and seniors. In addition, the reading test 
was given to MS 1 cadets, mostly freshmen. In SY 84-85, the English and the 
reading tests were administered to MS I and IV classes in the spring. The 
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mathematics test was admiaistered ia the Advaaced Camp during the summers of 
198A and 1985. In the last year of the research project (SY 85-86), data for 
English and reading were collected from MS I and MS IV cadets in the fall. 

This report is based on data from the first two years. Table 2 presents 
the data collection schedule, the military science classes included, and the 
number of cadets who participated in the testing. 

Tests were administered by the cadre in each ROTC detachment and then 



forwarded to the 
tapes were sent 


test publishers for scoring, 
to ARI from ROTC Headquarters 


The coded data on 
for analysis. 


magnetic 




Table 2 

ROTC Achievement Testing Program 
Testing Schedule and Number of Cadets Tested 




Tests and 
Time of Administration 


Class 
MS I MS III 


MS IV 


SY 83-84 


Spring Semester 








Nelson-Denny Reading 


20,218 






Missouri English 




7, 365 




Summer Advanced Camp 








Stanford Mathematics 


8,221 




SY 84-85 


Spring Semester 








Nelson-Denny Reading 


18,930 


7,306 




Missouri English 


16,001 


7,558 




Summer Advanced Camp 








Stanford Mathematics 


8,265 





Demographic Description of Cadets Participating in the Achievement Testing 
Program 

Tables 3 and 4 present the total number of cadets talcing each of the 
achievement tests during the two school years and the percentages of subgroups 
by ROTC region, gender, racial/ethnic groups, and academic major categories. 
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Table 3 describes the MS I students who took the English test in SY 84-85 
and the reading test in SYs 83-84 and 84-85. The total number of cadets 
included in each of these data sets represents approximately half of the 
end-of-school-year MS 1 enrollment for the respective years. The breakdown 
of cadets across ROTC regions reflect the regional distribution of ROTC 
detachments; most of the cadets attend schools in Regions I, II, and III. 
(See Appendix 1 for the states included in each ROTC region.) For the test 
data, the regional representation remained fairly stable over the two years 
with one exception of some decrease in cadets from Region III in SY 84-85. 

Cadets identified themselves into one of the following ethnic groups: 
(1) Asian/Pacific Islander, (2) black - Negro, not of Hispanic origin, 
(3) Hispanic, (4) white - Caucasian, not of Hispanic origin, (5) American 
Indian/ native Alaskan, and (6) Other. In the analyses and tables reported 
below, the American Indian/ native Alaskan group, representing about .5% of 
every data set, was combined with the Other group. In terms of racial/ethnic 
backgrounds, approximately 73% of MS X cadets were white, 21% black, 3% 
Hispanic, and 1.5% Asian/Pacific Islanders. For both school years, about 28% 
of the MS I cadets were females. 

The mathematics test was administered in Advanced Camp to cadets who had 
completed MS Hi classes and were to begin MS IV in the fall. For the sake of 
simplicity, the Advanced Camp cadets who took the mathematics test and the MS 
IV cadets who took the English and the reading tests will b« designated "MS 
IV" cadets in the remainder of this report. 

The MS IV data sets included much higher proportions of the end-of-school- 
year enrollment figures, ranging from 88 to 100% with a mean of 96%. The 
largest proportion of MS IV cadets tested came from Region I followed by 
Regions U, III, and IV, conforming to the regional distributions reported m 
the ROTC Enrollment Reports. Compared to the MS I data, the relative 
representation of the MS iV cadets from Region IV was considerably greater, 
while the relative representation of Region III cadets was smaller. 

Percentages of males and females in the MS IV data remained stable over 
the two school years at approximately 16% females and 84% males. However, 
compared to the MS I data, the relative proportion of females was reduced to 
nearly half, from about 28% to about 16%. This suggests that a large number 
of female cadets might not continue in the ROTC program at the contracting 
time. 

The racial/ethnic backgrounds of the MS IV cadets were: 73-76% white, 
16--18% black, 4-5% Hispanic, and 2% Asian-Pacific Islanders. The proportion 
of the black cadets was lower, and that of the Hispanics was slightly higher, 
than in MS 1 classes. 
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Table 3 

Demographic Composition of MS I Cadets Far tlclpa ting 
In the Achievement Testing Program (Percentage of Total Tested) 



Reading data Reading data English data 
Group SY 83-84 SY 84-85 SY 84-85 

N=20,218 N=18,930 N=16,001 



ROTC Region 








Region I 


32% 


36% 


36% 


Region II 


29 


32 


29 


Rppion III 


32 


25 


26 


Rpalnn IV 


7 


7 


8 


(la n H P T* 

V7CUUCX 








rcuIaJ.c 


27 


28 


28 


Male 


73 


72 


72 


Racial/Ethnic Group 








Asian 


1 


2 


2 


Black 


22 


21 


20 


Hispanic 


3 


4 


3 


White 


72 


73 


73 


Other 


2 


1 


2 


Academic Majors 








Engineer/ Archi tec t 


13 


12 


12 


Physical Science 


15 


14 


14 


Social Science 


12 


13 


12 


Medical 


7 


7 


8 


Other 


54 


55 


54 
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Table 4 

Demographic Composition of MS III and MS IV Cadets Participating 
in the Achievement Testing Program (Percentage of Total Tested) 



Reading Reading English English Math Math 
Group SY 83-84 SY 84-85 SY 83-84 SY 84-85 SY 83-84 SY 84-85 

N=7,524 N=7,306 N=7,365 N=7,558 N=8,221 N=8,265 



ROTC Regioa 














RegiOQ I 


43% 


44% 


42% 


43% 


39% 


43% 


Region II 


21 


25 






ZD 




Region III 


19 


17 


20 




9n 
zu 




Region IV 


16 


15 


16 


14 






Gender 














Female 


16 


16 


16 


17 


15 


15 


Male 


e4 


84 


84 


83 


85 


85 


Racial/Ethnic Group 














Asian 


2 


2 


2 


2 


2 


2 


Black 


18 


16 


18 


16 


18 


18 


Hispanic 


4 


6 


5 


4 


5 


5 


White 


75 


76 


73 


76 


73 


73 


Other 


1 


1 


2 


2 


2 


2 


Academic Majors 














Lngineer/ Archi tec t 


9 


12 


9 


11 


11 


12 


Physical Science 


15 


16 


14 


15 


15 


16 


Social Science 


32 


30 


31 


28 


28 


27 


Medical 


4 


4 


4 


5 


4 


4 


Other 


40 


38 


41 


41 


42 


41 



The proportions of cadets by academic majors are also indicated in Tables 
3 and 4. Academic majors were grouped, guided by the grouping procedure used 
by the Military Personnel Center (MILPERCEN) Officer Distribution Division as 
follows: Engineering/architecture, physical science (computer science, 
physical science, natural science, math/statistics), social science (social 
science, psychology), medical (pre-med, nursing), other (business/accounting, 
management, . education, letters, fine arts, and undeclared). 

The engineer/architecture group made up about 10% of MS I and IV classes. 
Likewise, the percentage of cadets in physical sciences was fairly consistent, 
at about 15% ot all classes tested. On the other hand, the percentage of 
social science majors differed; about 12% for MS I classes and about 30% for 
MS IV classes. The ''Other** group constituted the majority, 55% for MS I 
classes and about 40% for MS IV classes. The pre-med and nursing students 
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were combinev'i, even though their academic programs may differ substantially, 
w»ince together they make up less than 10% of the ROTC population (basic and 
advanced courses), and many of them are commissioned through the special 
appointment avenue* 

To summarize the demographic composition of the cadets participating in 
the achievement testing, the proportions representing the ROTC regions, 
genders, and racial/ethnic groups remained fairly constant over the two school 
years. Nearly the entire population of MS IV cadets participated. For MS X 
testing, the participation rate was a little better than 50%. However, for 
all of these classes tested, the proportional representation by region and 
gender closely approximated the percentage breakdowns reported by the ROTC 
Headquarters for the respective school years and MS classes. The differences 
between the sample subgroup percentages and the percentages reported in ROTC 
Closing Enrollment Reports ranged from 0 to 5 percent, averaging at 1.6 
percent. Comparing the demographic compositions between the MS I and IV 
classes, some marked differences were noted. The proportions of females and 
black cadets were lower in MS IV than in MS I. 



Results 

Data used for analysis were: the total score tor the Missouri College 
English Test, the total of the Vocabulary and Comprehension subtests for the 
Nelson-Denny Reading Test, and the Numeric Competence subtest score for the 
Stanford Achievement Test of Mathematics. 

The results of the analysis are presented in terms of the actual score a 
cadet achieved on each test (raw score) and the ROTC and national percentile 
scores. The raw scores are not comparable across the three achievement tests; 
the maximum possible scores for English, reading, and mathematics are 90, 172, 
and 43, respectively. Since a given raw score (e.g., 138 on the reading test) 
in itself does not denote a certain level of competence, one might refer to 
the percentile scores based on national samples. A percentile score indicates 
the percentage of people who scored at or below a given raw score. However, 
using the published percentile scores of these tests to infer ability levels 
may not be appropriate since the English and the mathematics tests were normed 
on test scores of college freshmen and high school seniors, respectively, both 
in the early 1960s. 

The ROTC percentile scores provide current, age-appropriate norms based on 
the same ROTC reference group across tests. Using the ROTC percentiles, it is 
possible to compare the relative positions of various subgroups, and also to 
compare individual performance across the three tests. For exc>mple, an 
English raw score of 37, a reading score of 92, and a mathematics score of 25 
all correspond to the 20th ROTC percentile score for MS IV cadets for the 
respective tests. This means that 20 percent of MS IV cadets scored below 
these raw scores on these tests. On the other hand, despite the limitations 
of the published national percentile scores, they do provide some basis on 
which to compare the ROTC percentile scores against the performance of 
students in general. For this purpose, both percentile scores are presented 
in some of the tables. 
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The score distributions were highly similar over the two school years, as 
exemplified in Table 5. For the remainder of the report, tabular and written 
descriptions of the results will be based mainly on the SY 83-84 data. The MS 
IV test results will be presented first, followed by the MS i test results. 

MS IV Achievement Test Results 

This section describes the results of the mathematics testing in Advanced 
Camp (in the suauner between the junior and the senior years) and the English 
and the reading testing of MS IV cadets in the spring. Analysis of these data 
was related to the issue of standards for contracting or commissioning. It 
was guided by the question of what scores might be used as selection criteria 
which would serve to retain the maximum aumber of cadets who possess 
sufficient basic skills to perform their duties as Army officers. The 
analyses were also directed at the differential effects that various cut 
scores might have on gender and ethnic subgroups. 

It must be emphasized here that the results reported below are not 
intended to suggest or recommend any selection standard. The particular 
ranges of scores examined in the following sections were chosen only for 
illustrative purposes, i.e., to show the patterns of score distributions for 
the whole group and for subgroups. 

Missouri College English Test . For this test, the maximum raw score is 
90, and the national norm is based on college freshmen of the early 60s. 
As shown in Table 5, the average raw score for the total group was 30 for both 
years. Taking the SY 83-84 data, the average for females was somewhat higher 
than for males, 54 and 49, respectively. However, more substantial 
differences were found among the racial/ethnic groups. The black and Hispanic 
groups scored substantially lower than the white group. 

Table 6 shows the number of cadets in these ethnic groups who scored at 
or above given scores on the English test. The figures in columns labeled n 
indicate the actual numbers of cadets scoring at or above a given score; the 
percentage in parentheses indicates the proportion of the subgroup (e.g., 
white, black) that scored at or above this score. The "total" column 
summarizes the results across the ethnic groups. 

A raw score of 37 on the English test corresponds to the 20th MS IV 
percentile point (meaning that approximately 80% of the total MS IV group 
scored at or above this score). For SY 83-84, 88% white, 47% black, 58% 
Hispanic, and 75% Asian cadets achieved a score of 37 or higher. In other 
words, 20% of the total MS iV cadets failed to achieve this score, but some 
ethnic groups were represented disproportionately in this 20%. 
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Table 5 

Average Raw aad Percentile Scores for 
Missouri College English Test, MS IV, SY 33-84 and SY 84-85 



SY 83-84 SY 84-85 









Percentile Scores 


Raw 




Percen 


Clle Scores 




Score 


•ie 

SD 




Nfl tin na 1 

LIO WXv/ lie* JL 


Score 


•ie 

SD 


ROTC 


Na tlonal 


To tal 


50 


15 


49.6 


40.2 


50 


15 


49.5 


40.4 


Geader 


















Male 


49 


14 


48.3 


38.9 


50 


15 


48.4 


39.3 


Female 


54 


15 


56.5 


46.7 


53 


15 


55.2 


45.7 


Ethnic Group 


















Asian 


48 


14 


44.8 


36.0 


49 


15 


46.2 


37.4 


Black 


38 


13 


26.5 


20.6 


38 


13 


26.0 


20.4 


Hispanic 


41 


15 


33.3 


26.3 


43 


14 


35.0 


27.8 


White 


54 


13 


56.8 


46.2 


53 


13 


55.6 


45.5 


Other 


46 


16 


42.9 


34.4 


48 


15 


45.6 


36.9 



^ Standard deviation of raw scores* 



Table 6 

Cumulative Number and Percentage of MS IV Cadets Scoring Above 
Various Scores on Missouri College English Test by Ethnic Group 



Itetcentile 
Raw 1 Scores 
ScorelROIC / Ifational 


WhltB 


Black 


Hispanic 


Asian 


Othsr 


Total 


n 


(%) 


n 


(%) 


n 1 


(%) 




;%) 




:%) 


n (%) 


SY 83 - 84 


























37 20 


15 


4737 


(88) 


629 


(47) 


213 


(58) 


116 


(75) 


101 


(70) 


5796 (79) 


34 15 


11 


4929 


(92) 


747 


(56) 


240 


(66) 


121 


(79) 


109 


(76) 


6146 (84) 


30 10 


7 


5116 


(95) 


914 


(68) 


270 


(74) 


D7 


(89) 


116 


(81) 


6553 (89) 


25 5 


4 


5269 


(98) 


1092 


(82) 


309 


(84) 


147 


(95) 


128 


(89) 


6945 (94) 


21 2 


2 


5323 


(99) 


1190 


(89) 


330 


(90) 


149 


(97) 


133 


(92) 


7125 (97) 


Tbtal 




5359 (100) 


1336 (100) 


366 (100) 


154 (100) 


144 (100) 


7359 (100) 



NOTE: The "Ibtal" is less than the total nunber tested dus to missing dam. 
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Nelson-Deany Read lag Test > The possible raw score range for this test is 
from 0 to JL72. However, the national norm provides a fairly accurate basis to 
interpret the raw scores relative to the performance of a national sample 
since the norms were developed for each college grade in early 1980s. 

The means for the total ROTC group remained stable over the two years at 
the raw score of 119. Table 7 shows that the average scores for males and 
females were very similar for this test, although large differences were found 
among the ethnic group means. 



Table 7 





Average 


Raw and Percentile 


Scores 


for 




Nelson-Denny Reading Test, MS 


IV, SY 


83-84 




Raw 




Percentile Scores 




Score 


SD* 


ROTC 


Na tional 


Total 


119 


30 


49.6 


43.0 


Gender 










Hale 


119 


30 


49.9 


43.3 


Female 


117 


32 


48.6 


42.4 


Ethnic Group 










Asian 


110 


30 


40.7 


34.3 


ElacR 


87 


26 


21.1 


16.0 


Hispanic 


96 


34 


30.8 


25.6 


White 


128 


24 


57.5 


50.8 


Other 


119 


32 


49.9 


43.9 



* Standard deviation of raw scores. 



Table 8 displays the number of cadets, by ethnic group, passing various 
score points. For SY 83-84, at the ROTC 20 th percentile (raw score of 92), 
91% of white cadets passed, but 39% and 55% of the black and Hispanic cadets, 
respectively, passed. The performance of ethnic minority groups was generally 
poorer, when compared to the total group, on the Nelson-Denny Reading Test 
than on the Missouri English test. 
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Table 8 

Cumulative Number and Percentage of MS IV Cadets Scoring 
Above Various Scores on Nelson-Denny Reading Test by Ethnic Group 



Bau 1 
Score} 






White 


Black 


Hispanic 


Asian 


Other 


Total 


Percentile 
R0IC 


Score.': 
fetlooal 


n 


(%) 


n 




n 


i%) 


n 


(%) 


n 


(%) 


£ (%) 


SY 83 -84 


























92 


20 


14 


5078 


(91) 


509 


(39) 


177 


(55) 


103 


(72) 


70 


(74) 


5977 (79) 


86 


15 


10 


5275 


(94) 


623 


(47) 


193 


(60) 


111 


(79) 


77 


(82) 


6315 (84) 


77 


10 


5 


5437 


(97) 


784 


(60) 


220 


(68) 


119 


(84) 


80 


(85) 


6677 (89) 


64 


5 


2 


5547 


(99) 


1031 


(79) 


252 


(78) 


129 


(91) 


88 


(94) 


7086 (94) 


54 


2 


1 


5591 


(99) 


1161 


(88) 


277 


(85) 


133 


(94) 


91 


(97) 


7294 (97) 


Total 






5607 (100) 


1312 (100) 


324 (100) 


141 (100) 


94 (100) 


7524 (100) 



Stanford Mathematics Test , As Table 9 shows, the overall raw score mean 
on this test was 32 (with the possible score range of 0 to 42). The national 
norms on this test are not very informative since they were developed in early 
60s based on performance of college-preparatory high school s^eniors. Males 
performed considerably better than the females on this test. Far SY 83-84, 
the respective ROTC percentile means were 51.3 and 39.4. Table 9 also shows 
the same pattern of ethnic group differences found with the other tests: 
blacks and Hispanics scored substantially lower than whites. 

Table 10 indicates that about 80% of the total ROTC group scored above a 
raw score of 25. However, this group included 87% of the whites, 84% oJt ttie 
Asians, 44% of the blacks and 61% of the Hispanics. 
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Table 9 

Average Raw and Percentile Scores for 
Stanford Math Test, MS III, SY 83-84 



Percentile Scores 

Raw Score SD ROTC National 



Total 


32.1 




8.1 








49.5 




43.2 


Gender 
Male 
Female 


32.6 
29.0 




8.0 
8.5 








51.3 
39.4 




44.9 
33.2 


Ethaic Group 
Asian 
Black 
Hispanic 
White 
Other 


33.1 
24.7 
28.1 
34.2 
30.9 




7.2 
7.6 
8.0 
7.0 
8.3 








52.4 
24.6 
35.3 
56.8 
45.1 




45.8 
19.0 
29.0 
50.4 
38.6 


* Standard deviation of raw scores. 














Table 10 

Cumulative Number and Percentage of MS III Cadets Scoring Above 
Various Scores for Stanford Mathematics Test by Ethnic Group 


Raw 1 Percentile Scores 
Score! ROIC Ibtiocal 


WhitE 
a (%) 


Black Hispanic 
n (%) n (%) 


Asian 
a (%) 


Other Total 
a (Z) a (Z) 


V/ 83-84 
25 20 14 


5231 


(87) 


663 (44) 


271 


(61) 


128 


(84) 


112 


(74) 6405 (78) 


23 15 10 


5456 


(91) 


796 (53) 


307 


(69) 


136 


(90) 


122 


^81) 6817 (83) 


20 10 6 


5712 


(96) 


1018 (68) 


356 


(81) 


140 


(93) 


132 


(87) 7358 (90) 


17 5 4 


5847 


(98) 


1228 (82) 


397 


(90) 


145 


(96; 


139 


(92) 7756 (94) 


14 2 2 


5924 


(99) 


1350 (90) 


422 


(95) 


150 


(99) 


144 


(95) 7990 (97) 


lotal 


5979 (100) 


1492 (100) 


442 (100) 


151 (100) 


151 (100) 8215 (100) 



Note: The "Ibtal" is less than tte total n'mber tested due to missing data. 
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Tbe gender and ethnic group scores across three tests > There were gender 
differences in two of the tests; females scored higher in English and lower in 
mathematics than ma^es. The average of males' English scores was not far 
below the total average, mainly because males constituted close to 85% of the 
total group* However, the females* average mathematics score was coa^iderably 
below that of the males* 

The ethnic minority groups, especially the blacks and the Hispanics, 
scored substantially lower than the white group in all skill domains tested. 
For all ethnic minority groups, the percentage of cadets passing the 20th 
percentile point was the lowest for the reading test. For black cadets, 39%, 
44%, and 47% passed the 20th ROTC percentile points of the reading, 
mathematics, and English tests, respectively. 

Analysis of composite scores based on three tests . The objective of the 
Achievement Testing Program was to determine basic skills qualification 
standards. However, die realities of officer production mission and the 
extent of differential impact of selection standards on subgroups must be 
considered in this process. An individual's performance on any given test can 
fluctuate, so that it is not judicious to base a critical decision concerning 
his or her future career on one test score. In addition, many individuals dv. 
not ha\ the same degree of competence in all skill areas, but rather perform 
better in one area than another. 

Table 11 shows the similarities of individual performance on two tests, 
e.g., between English and reading scores. The higher the correlations, the 
more similar the performance on two tests. For the total group, English and 
reading test scores are highly correlated (£ = .70), but the associations 
between English and mathematics and between reading and mathematics are only 
moderate ( r = .55 and .51, respectively). Many of the correlations between 
the verbal tests (English and reading) and mathematics are in the .40s range 
for ethnic subgroups. These results suggest that some cadets might score 
above the cut score (e.g., 40th ROTC percentile) on one test but below the cut 
score on another, largely due to varying expertise in different skill areas. 
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Table 11 

Correlations Between Achievement Test Scores 





English 


English 


Reading 




a uu 


aod 


and 




Reading 


Math 


Math 


To tal 


.70 


.55 


.51 


White 


.63 


.46 


.38 


Black 


. 61 


.45 


.40 


Hlspaalc 


.68 


.53 


.52 


Aslaa 


.78 


.48 


.48 


Other 


.71 


.77 


.63 



No te; ROTC perceatile scores were used for these analyses. Math data from 
SY 83 - 84, MS III; English and reading data from SY 84 - 85, MS IV 
classes. 



If several test scores are used to evaluate cadets' overall competence in 
basic skills, their strengths in one area may compensate for marginal scores 
in another. This effect is shown on Table 12, which displays (in the first 
three columns) the percentage of minority-group cadets passing the 10th and 
the 20th ROTC percentile scores for the three tests separately. The fourth 
column contains the percentages of cadets passing these points based on each 
cadet's average of the English and the reading ROTC percentile scores. The 
la?3t column shows the percentages based on averages of each cadet's ROTC 
percentile scores on all 3 tests. 
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Table 12 

Percentage of Mlaorlty-Group Cadets Exceeding the 10th aad the 20th 
ROTC Percentile Points for Each Test and Combinations of Tests 





English 

Uu.±y 


Reading 
unxy 


Math 
unxy 


and 
Keaaing 


and 

Mq rh 


and 

Mq 

fla tn 


All 


lutn iTcrccnuixc 




















U J /e 




/o 


i J /o 






Hispanic 


79 


63 


81 


83 


88 


83 


88 


Asian 


89 


81 


93 


93 


98 


95 


96 


20th Percentile 
















Black 


46 


40 


44 


48 


50 


49 


50 


Hispanic 


59 


46 


61 


66 


77 


67 


77 


Asian 


75 


67 


84 


75 


89 


86 


87 



Comparing the results for separate tests and those for averaged data 
Indicate that the number of minority cadets who would pass a given cut point 
would Increase If multiple test scores are considered together. For example^ 
If the 20th percentile score on the English test alone were used as a 
selection standard , only 46% of the black cadets would pass. If composite 
scores based on all tests , or English and mathematics, were used, 50% would 
pass. For the olack group, which constitutes about 20% of the total MS IV 
population, the difference of 4% or 5% currently amounts to 50-60 more cadets 
to be commissioned. 

Based on general knowledge about ability testing and the empirical 
Illustration made above. It seems that using a composite score for a selection 
procedure would have the following advantages: 

1) Multiple test scores ere more reliable than one to represent an 
Individual's overall abilities. 

2) Since many Individuals have different areas of strengths and since 
a basic levol of quantitative skills would be needed for many 
officer jobs, both skill areas should be assessed. 

3) If the English and mathematics scores are Included In the 
composite, the number of cadets passing the cut score Is likely to 
Increase. 



17 



ERIC 



4) By increasing the number of cadets passing, this approach would 
reduce the differential impact of selection process across the 
ethnic minority groups. 

However, currently there is no sound information on how well any of the 
separate tests or composite scores would predict performance in Aimy officer 
jobs. Analyses reported above suggest that additional research needed to 
clarify these questions is warranted* 

MS I Achievement Test Results 

The purpose for administering the English and the reading tests to MS I 
cadets was to examine the score distributions for new cadets which would, in 
turn, provide a basis for establishing remediation guidelines. As with the MS 
IV data, it was not possible to determine what level of reading and writing 
abilities a given test score represented since there was no other measure of 
these sk^ills with which the achievement test scores could be compared. 

Given this constraint, the following steps were taken: 1) compare raw 
scores of MS I and IV cadets and 2) examine the number of MS I cadets 
performing near the score which may be used as a cut score for selection 
(contracting and/or commissioning). Raw scores were used for comparison since 
percentile scores of MS I and MS IV groups are based on different reference 
groups and are not comparable. The following comparisons are based on SY 
84-85 data which included English and reading tests given to both MS I and IV 
classes. 
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Table 13 shows the average raw scores of MS I aad MS IV classes on English 
and reading tests for each class as a whole, by gender, and by ethnic group. 
The MS IV averages are consistently higher than the MS I averages* 



Table 13 
MS I and MS IV Raw Score Means 
(SY 84 - 85 Data) 



English Reading 





MS 


I 


MS 


IV 


MS 


I 


MS 


IV 


Total 


46 


(15) 


50 


(15) 


101 


(32) 


119 


(30) 




















Male 


45 


(15) 


50 


(15) 


102 


(32) 


120 


(30) 


Female 


48 


(16) 


33 


(15) 


99 


(32) 


117 


(32) 


Ethnic Group 


















Asian 


47 


(15) 


49 


(15) 


101 


(35) 


111 


(32) 


Black 


35 


(13) 


37 


(13) 


73 


(26) 


88 


(27) 


Hispanic 


37 


(16) 


43 


(14) 


76 


(35) 


93 


(34) 


White 


50 


(14) 


53 


(13) 


110 


(28) 


128 


(24) 


Other 


43 


(16) 


48 


(15) 


95 


(34) 


121 


(33) 



Note: Score range for the English test is 0 - 90. 

Score range for the reading test is 0 - 172. 
Standard deviations are indicated in parentheses* 



Table 14 displays the percentages of MS I cadets scoring at or above a 
hypothetical cut score for MS IV cadets. The raw scores of 37 and 93, for the 
English and the reading tests, respectively, were the scores at which 20% of 
MS IV cadets in SY 84-85 failed. This table also shows a smaller percentage 
of MS I csdets passing these hypothetical cut score points on both tests, 
across ethnic groups. 
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Table 14 

Percentages of MS I aad MS IV Cadets Passing a Given Score 
on Missouri English and Nelson-Denny Reading Tests 
by Ethnic Group (SY 84 - 85) 



Raw Score White Black Hispanic Asian Other Total 

English 

MS I 37 80% 38% 46% 71% 61% 69% 

MS IV 37 87% 46% 59% 75% 72% 78% 
Reading 

MS I 93 71% 20% 31% 55% 50% 58% 

MS IV 93 90% 40% 46% 67% 75% 79% 



Note: English raw score of 37 and reading raw score of 93 are the scores that 
about 80% of MS iV cadets passed in SY 84-85. 



It is difficult to determine why the MS IV group scored higher than the MS 
I group, especially since the data for the two groups are not longitudinal 
(i.e., not collected from the same group of people at (wo points in time). 
Part of the difference may reflect attrition of low achievers from colleges 
and ROTC programs between MS I and MS IV years. The attrition may be 
voluntary or due to selection applied at contracting time. The difference may 
also reflect more years of education and greater expertise on the part of the 
older cadets. 

If these changes are the primary reasons for the score differences between 
MS 1 and MS IV cadets and if ROTC were commissioning the required number of 
new officers each year, then a special effort for remediation training may not 
be necessary. However, currently there is a need to increase the number of 
commissionees. If remediation programs could improve the basic skills of 
marginally achieving cadets, it would enlarge the pool of cadets available for 
commissioning. 

An approach for establishing remediation guidelines > As mentioned 
earlier, there are no data available to determine the relationship between the 
achievement test scores and performance in officer jobs. In addition, since 
the Achievement Testing Program data are not longitudinal, the degree to which 
MS I students might increase their basic skills without special remediation 
training is not known. Given this situation, the only feasible way to develop 
preliminary guidelines for early remediation might be to use scores which 
might be chosen as selection criteria as reference. 
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As an illustrative example^ suppose that a cut score is set at the raw 
score at which 80% of MS IV cadets pass* (Depending on the size o£ cadet pool 
and officer production requirements^ the cut score might be higher or lower*) 
Even though the overall performance of the MS 1 cadets was lower than that of 
MS IV classeS) it is assumed that by the time of selection (for contracting or 
commissioning), a cadet would be expected to pass the cut score point* It is 
also assumed that those who scored somewhat above the cut score at time 1 (say 
in MS I class) may not pass at a later time, since any individual's 
performance on a given test can fluctuate over multiple testing* Further, 
those who barely make the cut score would benefit from developing the basic 
skill* On the other hand, those who score somewhat lower than the cut score 
may also increase their skills through special training and pass the test the 
second time. 

The question is what proportion of MS I cadets fall in the score range of 
"somewhat above" and "somewhat below" the cut score* For English and reading 
data from SY 84-85, the MS I cadets were grouped into these score categories, 
using as the "cut score point" the score where 80% of MS IV cadets passed* 
Table 15 shows the percentage breakdowns by test* 



Table 15 

Percentage of MS I Cadets in Score Ranges 
Close to a Possible Cut Score (SY 84 - 85) 



Raw Score Range Percent 



English 

44 and above 57 

37 - 43 15 

30 - 36 12 

29 and below 17 

Reading 

101 and above 50 

93 - 100 9 

80 - 92 14 

61 - 79 15 

60 and below 12 



Note: English raw score of 37 and reading raw score of 93 are the 
scores that 80% of MS IV cadets passed in SY 84-85* 
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For English, 80% of MS IV cadets passed the raw score of 37 (the MS IV 
average was 50), About 15% of MS I cadets scored 37-43. About 12% ot MS I 
scored 30 to 36. A score of 30 is at about the 10th percentile point for MS 
IV cadets and would be an extremely low score as a selection standard* Of 
course, students falling in the very low score category may possibly improve 
skills to pass the cut score, with good instructions and high motivation. But 
it appears particularly beneficial for cadets in the middle groups (somewhat 
above and below the cut point) to be strongly encouraged to seek remedial 
training. 

For the Nelson-Denny reading test, the hypothetical cut score, based on 
80% pass for MS IV Cadets (SY 84-85), Is 93, The MS IV average is 119. The 
score range that might be considered for remediation training is from 80 to 
100, About 23% of MS I cadets fall in this middle range • On this test, about 
12% of MS I cadets scored 61 or below, which is the range where about 5% of MS 
IV cadets scored • It may be difficult for most students scoring in this range 
to improve their reading skills enough to pass 92, 

Comparison of Average Scores by Academic Majors 

Table 16 presents average raw and ROTC percentile scores for MS 1 and 
MS IV cadets by test and by academic major groups (i.e., engineering/ 
architecture, physical science, social science, medical, and oUier). Note 
that raw score averages allow comparisons between MS I and IV cadets, while 
the ROTC percentiles indicate the position of a given subgroup relative to the 
total MS class. The means of MS IV class were based on data from SY 83-84 and 
SY 84-85 classes combined. 
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Table 16 

Average Raw and ROTC Perceatile Scores by College Major 



Engineer/ Physical Social 
Architect Science Science Medical Other 

Missouri English 

MS IV (10) (15) (30) (04) (41) 

Raw Score 52 52 51 54 48 

ROTC Percentile 53 54 50 57 46 

Nelson-Denny Reading 

MS IV (10) (16) (31) (04) (39) 

Raw Score 121 121 122 125 114 

ROTC Percentile 52 52 53 55 44 

Stanford Math 

MS III (11) (16) (27) (14) (42) 

Raw Score 37 34 30 33 31 

ROTC Percentile 68 57 43 55 45 

Missouri English 

MS I (12) (14) (12) (08) (54) 

Raw Score 50 48 48 49 44 

ROTC Percentile 57 53 53 54 45 

Nelson-Denny Reading 

MS I (13) (15) (12) (07) (54) 

Raw Score 110 103 106 103 96 

ROTC Percentile 58 52 54 52 45 



Note: Numbers in parentheses indicate percentage of each major group in 
respective MS class* 

Medical category includes pre-med and nursing majore 



For both MS 1 and IV classes, average scores on the English test were 
fairly close among the engineering, physical, and social science groups; the 
group averages for all of these groups were 50th or above ROTC percentiles. 
The "Other" group, which makes up the majority, scored lower. For MS IV 
cadets, the reading averages of all groups except "Other" were very close, and 
substantially higher than the "Other" average. This pattern is generally 
repeated for the MS I data. 
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The mathematics test was given only to MS III cadets, and no basic and 
advanced course comparison was possible. However, this test revealed more 
striking differences among the academic major giroups. Engineering/ 
architecture students scored, as a group, substantially higher than the 
physical science majors, who in turn scored higher than the social science and 
**0t>3er'* groups. 

Recall that the main difference in the academic major compositions between 
MS I and IV classes was that the percentage of social science majors was 
greater in MS IV than in MS I. In MS IV, this group performed as well as the 
engineering and physical science groups on English and reading tests. 



Summary and Conclusions 

The ROTC Achievement Testing Program was designed to serve two purposes: 

1) to establish guidelines for diagnosing weaknesses in basic skills and 

2) to establish selection standards. 

There were several constraints in the research design. Only achievement 
test data were collected, with no other measure of academic achievement to 
verify the meaning of scor'^s on these tests. (3oupled with outdated and/or 
age-inappropriate national norms, interpretation of the "goodness** of any 
given score was difficult. Different year groups of cadets were involved in 
testing each year so that changes from MS I to MS IV years could not be 
inferred. 

Some of the major findings from the Achievement Testing Program are 
suDunarized below* However, with the lack of many other essential data, the 
findings reported in this report must be considered preliminary and partial 
with reference to the issues of diagnostic and selection standards. 

ROTC norms enabled comparisons of achievement levels among subgroups. For 
both MS I and IV classes, gender differences were relatively minor, but there 
were considerable disparities among ethnic groups across all tests. Using 
composite scores, derived from averaging two or more test scores, may provide 
a way to assess overall achievement level across various academic domains, to 
increase some cadets* chances of meeting the selection criteria, and to 
contribute towards ROTC officer production mission. 

About 20% of MS 1 cadets scored somewhat above or somewhat below a score 
that was arbitrarily selected as a possible cut score for selection. If these 
cadets are informed of selection standards early in their college career and 
seek additional training in their weaker areas, this may have a significant 
impact on the overall quality and the number of ROTC- trained commissionees. 

Analyses of academic majors showed that the majority of cadets major in 
subjects other than engineering/architecture, physical, and social sciences 
and that they, as a group, scored lower than these groups. The engineering/ 
architecture and the two science groups did not differ greatly in reading and 
writing skills, though the social science group* scored lower than the other 
two groups in mathematics* 
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Ensuring the basic competency of new officers Is an essential mission of 
all precommlsslonlng programs* However ^ the Army ROTC also has a mission to 
increase the number of commlssionees during the next several years when a 
decline is expected in college enrollment. For example, ROTC projects 
commissioning 7,500 new officers in FY87 with the requirement of 9,200 (U.S. 
Department of the Army, 1986, in preparation). £ven with a basic skills 
selection standard which disqualifies the lowest 10% of the prospective 
commlssionees, the rate of production shortfall would increase from 18% to 
27%. 

In order to approach systematically the multiple objectives of ROTC, 
additional empirical research is needed. First, there must be a clear 
knowledge of which, and how much, basic skills are needed for Army officer 
jobs. Then, the effectiveness of basic skills development programs needs to 
be assessed. Issues to be addressed include: Which basic skills are amenable 
to developmental procedures? Which method is most effective and 
cost-effective? What levels of deficiencies are most likely to be improved? 
What are the rate and permanence of change? 

One such effort was undertaken within the military context during the 60s 
and early 70s. The focus of this project was on functional literacy among the 
enlisted personnel and effectiveness of an experimental reading training 
program which was integrated into the military-job-skills training. In terms 
of job-related reading skills, a substantial improvement was obtained (Sticht, 
1975a, 1975b). 

Unfortunately, these findings are not directly applicable to college 
population, curriculum, and officer job requirements. In addition, although 
basic skills remediation programs are commonly offered at most colleges and 
universities, systematic information concerning their effectiveness is scarce. 
Thus, examining the effectiveness of developmental procedures seems 
warranted. 

Finally, decisions regarding selection standards also must be based on 
officer job analysis and knowledge of predictive relationships between various 
basic skills measurements and officer job performance. 
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APPENDIX 1 



ROTC Region Jurisdictioa 



First ROTC Region, Fort Bragg, NC 2b307 



Connecticut Maine 
Delaware Maryland 
District of Columbia 



Florida 
Georgia 



New Hampshire 
New Jersey 



New York 
North Carolina 
Massachusetts 
Rhode Island 
South Carolina 



Vermont 
Virginia 
Pennsylvania 
Puerto Rico 
Virgin Islands 



Second ROTC Region, Fort Knox, KY 40121 



Illinois 
Indiana 



Kentucky 
Michigan 



Missouri 
Ohio 



Tennessee 
West Virginia 
Wisconsin 



Third ROTC Region, Fort Riley, KS 66442 



Alabama 
Arkansas 



Louisiana 



Mississippi 
New Mexico 



Oklahoma 
Texas 



Fourth ROTC Region, Fort Lewis, WA 98433 



Alaska 

Arizona 

California 

Colorado 

Hawaii 



Idaho 
Iowa 

Minnesota 

Montana 

Nebraska 



Nevada 

North Dakota 
Oregon 
South Dakota 
Utah 



Washington 

Wyoming 

Guam 

American Samoa 
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